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Abstract 

To design a series of assessments that could be used to compare the learning gains of high 
school students studying the cardiopulmonary system using traditional methods to those who 
used a collaborative computer simulation, called “Mr. Vetro”. Five teachers and 264 HS 
biology students participated in the study. The students were in regular biology, advanced 
biology, anatomy and physiology or AP biology. One group of teachers taught a 7 day unit on 
the cardiovascular system using Mr. Vetro and the other group of teachers taught the same 
content without Mr. Vetro. Assessments were administered before and after the unit, and 
included basic recall as well as making connections and application of knowledge to new 
situations. The learning gains of four matched Vetro vs. Comparison groups were compared, 
for all types of assessments. Results indicate that there were statistically significant differences 
between the learning gains of all of the Vetro vs Comparison groups for assessments that 
required higher level thinking skills as well as for some of the assessments that required basic 
recall. Learning gains on multiple choice problems taken from standardized tests were more 
moderate or not significant. Results show that students who use the Mr. Vetro collaborative 
computer simulation make stronger learning gains than those who do not, regardless of the level 
of the course. Schools should consider using the Mr Vetro simulation to foster a deeper 
understanding of the cardiovascular system. (Contains 2 Figures and 1 Table). 

Objective 

To design a series of assessments that could be used to compare the learning gains of HS 
students studying the cardiopulmonary system using traditional methods to those who used a 
collaborative computer simulation, called “Mr. Vetro” (Ioannidou, et.al, 2010, Repenning, et. al, 
2005). 

Conceptual Framework 

We predicted that students in the Mr. Vetro group would show higher achievement gains than 
those in the comparison group, because the computer simulation promotes collaboration between 
students as they control Mr. Vetro ’s organ parameters (Ioannidou, et.al, 2010, Repenning, et. al, 
2005). The simulation demonstrates the relationship between the cardiovascular system and 
pulmonary system that is difficult to demonstrate otherwise. 

Assessments included three “Levels of Thinking” (Shafer and Foster, 1997): 

• Level I questions assess the recall of basic facts and/or development of skills. 

• Level II questions involve making connections and/or integrating of information. 

• Level III questions involve analyzing, interpreting and/or applying knowledge in new 
situations. 
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Methods 

Both groups had between 5 and 7 days of instruction on the cardiopulmonary system (Luhn, 

2010 ). 

The Mr. Vetro treatment group was comprised of three teachers and 169 students in nine classes. 
The Mr. Vetro treatment group used the computer-based simulation with related activities. 
Some examples of simulations were an elite athlete and a smoking Mr Vetro exercising and a 
“normal” Mr Vetro overdosing on caffeine or valium (Luhn, 2010). 

The comparison group was comprised of three teachers and 95 students in five classes. The 
comparison group used traditional methods for teaching the same learning objectives. Some 
examples of activities were students exercising and then measuring their own heart rate, and 
traditional lecture-style lesson (Luhn, 2010). 

Students in both groups completed written assessments just prior to and just after the unit. 

Data Sources 

The Level I questions in the written assessments were selected response, marked as correct (1 pt) 
or incorrect/no response (0 pts). Level I questions included selecting typical values for “at rest” 
heart and lung parameters and oxygen saturation (i.e., heart rate, breathing rate). They also 
included multiple-choice questions that were collected from various conventional assessments 
(AP Biology Exam, NY State Regents Exam, Programme for International Student Assessment 
(PISA), Trends in International Mathematics and Science Study (TIMSS), National Assessment 
of Educational Progress (NAEP), and the Scholastic Achievement Test (SAT). The multiple 
choice questions were chosen as best fit to the curriculum. 

The Level II & Level III questions were open response, and were graded on 0-4 point rubric 
developed by design team. The Level II questions were definitions of vocabulary words used 
throughout the unit (i.e., hyperventilation, oxygen saturation, hypoxia, etc). The Level III 
questions, which were only administered in the post-assessment, involved applying knowledge 
learned in the classroom to a novel scenario, (i.e., what happens to parameter when mountain 
climber exercises at altitude or when a person has coronary artery disease?) 

Results 

Only students who took both the pre-assessment and the post-assessment are included in the 
data. Independent two-tailed t-tests and effect sizes were calculated. 

Aggregate Comparison. An aggregate comparison was made between the learning gains 
(difference between pre and post assessment results) of the entire Mr. Vetro group (n =169) vs. 
the entire comparison group ( n =95). See Ligure 1. In all Levels of assessments, the learning 
gains of the Mr. Vetro group were significantly greater than those of the comparison group, 
(multiple choice, t(205) = 1.97, p < 0.001; typical values, t(226) = 1.97, p < 0.001; vocabulary, 
t(l 84) = 1.97, p < 0.001). Respective effect sizes were 0.42, 0.99, and 1.42. The Mr. Vetro group 
differed significantly from the Comparison group in the multiple choice pre-assessment scores. 
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There were more advanced students in the Mr. Vetro group as a whole. 



Figure 1: Aggregate Results, Learning Gains of Mr. Vetro vs. Comparison 
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Comparison of matched groups. Four matched groups of Mr. Vetro vs. Comparison classes 
were then selected for further analysis. For the four matched groups, differences in responses to 
Level III tasks were statistically significant in favor of Mr. Vetro (Group 1: p < 0.05, Groups 2- 
4: p <.001 ). Respective effect sizes were 0.69, 0.76, 1.04 and 0.98. See Table 2 and Figure 2. 
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Table 1: Description of 


Matched Vetro vs. 


Comparison Groups 


Group 

# 


Description 


Group 

Type* 


Teacher 


School 


#of 

Classes 


#of 

students 


Type of 
Class 


1 


Same teacher at 
same school 
teaching same 
course 


V 


A 


A 


One 


17 


Regular 

Biology 


C 


A 


A 


One 


12 


Regular 

Biology 


2 


Two different 
teachers from 
different schools 
teaching same 
course 


V 


A 

B 


A 

B 


One 

One 


44 


Regular 

Biology 


c 


A 

C 


A 

C 


One 

Two 


57 


Regular 

Biology 


3 


Two different 
teachers from 
different schools 
teaching same 
course 


V 


D 


C 


Two 


29 


Anatomy & 
Physiology 


c 


E 


B 


Two 


38 


Anatomy& 

Physiology 


4 


Two different 
teachers from same 
school teaching 
advanced students 
(mixed class type) 


V 


B 


B 


Three 


85 


2 AP Bio, 1 
Adv Bio 


c 


E 


B 


Two 


38 


Anatomy & 
Physiology 



V = Mr. Vetro Group, C = Comparison Group 



Figure 2: Post- Assessment Results of Level III questions, Matched Mr. Vetro vs. 

Comparison Groups 
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The learning gains of the responses to the Level I multiple choice questions were not notable. 
Groups 1 and 3 had no statistical difference and had very small effect sizes. The other two were 
statistically significant (p<0.05 ) and the effect sizes were small to moderate (0.44 and 0.58). 

The learning gains of the responses to the Level I typical values questions were statistically 
significant in favor of Mr. Vetro for all matched groups (Groups 1, 3 and 4: p<0.001, Group 2: 
p<0.05 ). The effect sizes were large to very large (0.65 to 1.06). 

The learning gains of the responses to the Level II vocabulary questions were statistically 
significant in favor of Mr. Vetro for all matched groups (Groups 2-4: p<0.001, Group 1: p<0.05 ). 
The effect sizes were large to very large (0.81 to 1.60). 

Conclusions/Significance 

The comparison of pre- post-unit responses to the assessments demonstrates that the Mr. Vetro 
tech-based collective simulation approach, for similar content, results in stronger learning gains 
for all groups regardless of the level of the course (e.g., regular or honors). Statistically greater 
gains occurred in the results of Level I recall of typical values of the heart and lung parameters 
and in Level II vocabulary definitions. Differences in the gains in Level I recall of multiple 
choice questions from conventional assessments were more moderate, or in some matched 
groups, not significant. It is notable that responses to post-assessment Level III questions, which 
require critical thinking and application, were significantly more complex in the Vetro groups 
than those in the matched Comparison groups. 

The Mr. Vetro collaborative computer simulation shows promise as a tool for students to have 
steeper learning gains of the cardiopulmonary system. 
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